System and method for rapid and accurate histologic analysis of tumor margins using machine learning

ABSTRACT

This invention provides a histologic system and method for rapidly and accurately assessing tumor margins for the presence or absence of tumor using machine learning algorithms. This affords a rapid and accurate histologic tumor readout and increase process efficiency and decreases the chance for human error. Advantageously and uniquely, the system and method allows for analyzing the tissue section as complete or incomplete as the first criteria to determine whether a tissue section is clear of tumor. A machine learning process receives whole slide images (WSI) of tissue and determines (a) if each image of the WSI contains complete/incomplete tissue samples and (b) if each image of the WSI contains tumorous tissue or an absence thereof. A reconstruction process generates a model of the tissue that maps types of tissue therein, and a display process provides results of the model or report for use and manipulation by a user.

FIELD OF THE INVENTION

This invention relates to systems and method for analyzing images oftissue samples for medical and research diagnostic purposes, and moreparticularly to those employing machine learning and/or neural networksoperating on computing systems.

BACKGROUND OF THE INVENTION

Successful treatment of solid cancers relies on complete surgicalexcision of the tumor for either definitive treatment or prior toadjuvant therapy. Incomplete excision of tumors results in both localtumor recurrence and increased risk of distant metastasis and decreasedsurvival. Recurrence rate of tumors following surgical excision rangefrom as high as 50% for subtypes of brain cancer to as low as 0.5% forskin cancers. Tissue preparation/grossing, inking, grossing, andanalysis may contribute to this wide range in local recurrence rates.Varied methodologies can be used to gross and ink the tissue prior totissue sectioning and then analysis by the pathologist. The“breadloafing” methodology is the most commonly used technique forgrossing tissue, but in its standard use, only analyzes approximately 1%of tissue margins with this number decreasing as the size of tissueincreases. An advantage of the standard breadloafing methodology is thatit is quick and does not require the pathologist to analyze a largeamount of tissue. A less common methodology entails the use of en facemargins or Mohs Micrographic Surgery, in which 100% of the peripheraland deep margin is analyzed. A caveat to en face margins is that theperipheral margins are separated from the deep margin resulting in thepotential for lost information at this point. Mohs Micrographic Surgery(MMS) places relaxing cuts into the tissue allowing the peripheralmargins to fall into the same plane as the deep margin allowing true100% margin analysis without separation of the peripheral and deepmargin. MMS works with fresh tissue that is rapidly frozen and sectionedresulting in tissue being ready for histologic analysis in approximatelytwenty minutes. Both en face margins and MMS result in very low localrecurrence rates while also limiting the amount of healthy surroundingtissue removed. In cases of skin cancer, use of the above procedures, incritical anatomic areas, maintains function and facilitatesreconstruction that restores normal human appearance. Note that the useof the breadloafing technique in its current form, and permanent sectionhistology, results in taking significantly larger margins to increasethe likelihood of complete tumor resection as it is not performed inreal-time. Increasing the number of pieces generated and sectioned in abreadloafed model can increase the percentage of margin analyzed thoughthis is time consuming both at the generation step and the pathologicreading step. Breadloafed sections do not have to be limited to paraffinembedded sections but could also be done using frozen tissue in order toallow it to happen in real-time.

Challenges in en face and MMS margin analysis because of the relianceupon the requirement of complete tissue sections without missingperipheral or deep margins. Complete tissue sections are critical, asholes in the tissue may have resulted from tumor “falling out” of thetissue during tissue processing. Acquiring complete tissue sections canbe challenging depending on the type and size of the tissue. Therefore,during histologic analysis of the tissue, in addition to the presence orabsence of tumor at the margins, the pathologist must also evaluate thetissue to ensure that it is complete. This issue also arises in thesectioning of permanent paraffin sections, but less frequently.Challenges with en face margin and MMS margin analysis is that, as thesize of the tissue increases, the surface area or total margin to beanalyzed increases exponentially. The time of tissue section analysis bythe pathologist is directly related to the size of the tissue section onthe slide and the number of slides to be analyzed which are both relatedto the size of the tissue specimen. Furthermore, it is not uncommon inen face or MMS margin analysis that the positive margin consists of lessthan 1% of the total tissue margin. This places significant stress onthe pathologist to scrutinize the margin to ensure that it is adequatelyanalyzed to prevent missed tumor. During removal of the tissue, both theremaining defect and tissue removed is oriented and tagged to ensureaccurate identification of positive margins for subsequent tissueresection. Histologic analysis of the tumor results in a tissue “map” inwhich the presence of tumor is noted (annotated) on the tumor map.Depending on the size of section analyzed and the number of tissuesections this can be challenging to accurately and efficiently map thetumor. Subsequent resection is reliant on precise mapping and failure toaccurately map the tissue results in “clear” tumor margins when, inreality, it is an incomplete tumor resection. Finally, the informationmust be communicated to the operating room, informing the surgeon of thelocation of the remaining tumor. Currently, in the operating room thisis done by telephone, and in MMS a physical paper copy of the map iscarried back to the operating room. Communication via telephone requiresstaff or the surgeon to recreate the tumor map either mentally orphysically presenting the opportunity for mistakes. Alternatively, iftissue is being processed from a standard excision with post-operativemargin analysis, the information must be communicated to the surgeon viapathology report.

Tissue processing, staining, histologic analysis, and tumor mapping allrequire time. In use cases of MMS, local anesthesia is used and thepatient is placed in a nearby waiting room. In cases of tumor resectionin the operating room, the patient is under general anesthesia. Notsurprisingly, the risk of adverse events increases the longer a patientis under general anesthesia. Design of efficient and accurate systemsaimed at decreasing tissue analysis and processing time, and increasingthe speed at which information is communicated to the operating room,are highly desirable, and should desirably allow en face/MMS marginanalysis or increased numbers of breadloafed sections to be performed inany setting—thereby resulting in lower recurrence rates, less tissueresection, and increased efficiency and cost-effectiveness of care.Additionally, decreasing the time required by the pathologist to readpermanent sections from a standard excision will allow the pathologistto spend time in other areas of their practice or allow them to readmore specimens on a daily basis.

In addition to clinical medicine, there is also an unmet need in basic,translational, and clinical research for efficient and accuratemechanisms of tumor margin analysis. Understanding the efficacy of aspecific medical or surgical treatment in both animal models and earlyhuman studies rely on effectively evaluating tissue for the presence orabsence of tumor. As stated above analysis of more tissue requires moretime and resources which can be greatly decreased by the use of machinelearning to facilitate this process. This may allow increased numbers ofsamples to be analyzed thereby increasing the power of the study. Justas importantly, in research settings analysis of tumor margins may beperformed by individuals including students who are not experts in thefield. Access to a machine learning algorithm developed and trained withwhole slide images annotated by board certified and expert pathologistswill aid in accurate analysis of tumor margins thereby increasing thevalidity of animal and human early stage studies.

SUMMARY OF THE INVENTION

This invention overcomes disadvantages of the prior art by providing asystem and method for rapidly and accurately assessing tumor margins forthe presence or absence of tumor using machine learning algorithms thatcan be used in either clinical or research settings. If a tumor ispresent then the system and method allows for precise tumor mapping andaids the surgeon in planning for subsequent rounds of tissue resectioneither in real-time or a staged setting. The ability to automate theabove process and provide a rapid and accurate tumor readout increasesprocess efficiency and improves patient outcomes. Advantageously anduniquely, the system and method allows for analyzing the section ascomplete or incomplete as the first criteria to determine whether atissue section is clear of tumor.

In an illustrative embodiment a system and method for generating a modelof tissue for use in diagnostic and surgical procedures in clinical andresearch settings is provided. The system and method includes a machinelearning process that receives whole slide images (WSI) of tissueremoved from a patient or research (e.g.) animal model, and thatdetermines (a) if each image of the WSI contain complete or incompletetissue samples and (b) if each image of the WSI contain tumorous tissueor an absence of tumorous tissue. A reconstruction process generates amodel of the tissue removed from the patient with a mapping of types oftissue therein, and a display process provides results of the model foruse and manipulation by a user and/or a pathology report that can beadapted for uploading to an interested party. Illustratively, themachine learning process is trained using a plurality of training slideimages having a plurality of differing tissue types and arrangements.The training slide images can be provided via at least one of a libraryof preexisting images of tissue from third parties and preexistingimages generated and stored by the user. The system and method canfurther include a background removal process and a connected componentprocess that pre-processes each of the slide images prior to performingthe machine learning process. The model can define differing sections ofthe tissue based upon each of the slide images and components of thetissue within each of the sections, and/or include an infographic thatis arranged to allow the user to access further information or imageswith respect to the sections or the components of the tissue via a userinterface. Illustratively, the infographic is arranged to display aplurality of colors or other graphical representation that correspond todifferent types of tissue. Additionally, the machine learning processcan reside, at least in part, on a remote server accessed by the uservia a network. As such, an access control process can be used toauthenticate the user relative to the remote server and a billingprocess can generate financial transactions between the user and anoperator of the system with respect to operation of the system. Invarious embodiments, a whole slide imager is arranged to read each WSIprepared by the user and provide the slide images therefrom to themachine learning process. The slides associated with the WSI can beprepared using frozen sections or paraffin embedded sections of thetissue. Additionally, the tissue can be sectioned using an MMS, en faceor breadloaf (radial) technique. In general, the pathology report caninclude a readout that is adapted to be uploaded to a chart of thepatient.

In an illustrative embodiment, a medical treatment method performing theaforementioned system and method can be employed in whole or in part bya user. Such treatment method can be performed entirely by a single useror distributed amongst a plurality of entities or individuals.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention description below refers to the accompanying drawings, ofwhich:

FIG. 1 is a diagram of an exemplary breadloafing process for cuttingtissue slices from excised tissue, for use in creating slides that canbe imaged in accordance with the system and method herein;

FIG. 1B is a diagram of the of the application of the system and methodherein to both clinical and research settings, including but not limitedto various tissue grossing techniques, embedding media and tissueprocessing approaches, timing of machine learning algorithm/processanalysis, and output of an infographic and/or a report;

FIG. 2 is a schematic diagram of an overall hardware and softwarearrangement in which the illustrative system and method operates,according to an exemplary embodiment;

FIGS. 3A and 3B are two parts of an overall flow diagram showing thevarious procedure steps in an overall workflow of the system and methodof FIG. 2;

FIG. 4 is a plow diagram showing the processing of whole slide images(WSI) by the various processes of the system and method of FIG. 2;

FIGS. 5 and 6 are images of slides associated with a complete versusincomplete analysis using the COMP_NET machine learning processaccording to the system and method of FIG. 2;

FIGS. 7 and 8 are images of slides associated with a positive ornegative (e.g.) basal cell carcinoma (BCC) analysis using the BCC_NETmachine learning process according to the system and method of FIG. 2;

FIG. 9 is an image of a heat map of a tissue sample constructed basedupon the results of COMP_NET and BCC_NET according to the system andmethod of FIG. 2; and

FIG. 10 is a schematic diagram showing the runtime model for generatingand displaying an infographic for use by a practitioner (pathologist,surgeon, etc.) from tissue resected/excised from a patient according tothe system and method of FIG. 2.

DETAILED DESCRIPTION I. Background Considerations

FIG. 1 shows a generalized diagram of an exemplary excised tumor tissuesample 100—for example a skin lesion—that has been subjected to thebreadloafing technique, by way of non-limiting example. In thistechnique, the excised tissue sample 100 is sliced laterally, as shown,into sections (A-D), from which are taken thin slices 110, 120 and 130at various points along the length of the tissue sample 100. Thedistribution of slices 100-130 through the tissue is chosen to interceptthe likely extent of the cancerous growth 140 therein. Hence in slice120, there resides a piece 142 of the overall cancerous growth 140.Based upon one or more slides (described below) that are created fromthe slices, the pathologist should be able to determine the boundariesof the cancerous tissue, thereby ensuring that it is completely removedby the excision process.

However, using conventional techniques, it is possible that irregularlyor unpredictably shaped tumor boundaries may be overlooked by theprocedure—as indicated by the missed cancer tissue 150 on the edge ofthe section B. As described above, this is an undesirable outcome thatcan result in recurrence and/or spread of the cancer. The followingdescription can apply to the (a) MMS (b) en face, (c) breadloafed tissuesections generated from both clinical and research settings, and/orother applicable techniques that exist or can be developed by those ofskill in the art.

By way of further overview, FIG. 1B is a diagram 150 showing thesettings 150, including a clinical setting 154 and a research setting156 to which the machine learning algorithm/process described below canbe employed/applied to analyze tissue specimen margins for the presenceor absence of tumor. It shows that the algorithm/process can be appliedto tissue 158 grossed in any form of grossing method/technique 160including but not limited to MMS 162, En Face 164, Breadloafed sections(e.g. radial) 166 and embedded (170) in either OCT for frozen sections172 or paraffin for permanent sections 174. The algorithm/process 176can then be applied to these embedded specimens (170). Moreparticularly, the algorithm/process 176 is applied at various times(180), that is, in real-time for intraoperative margin assessment 182which would most commonly employ frozen sections, or in post-operativemargin assessment 184, which would most commonly employ paraffinembedded sections. The algorithm/process 176 can also be applied in aretrospective analysis 186, after a procedure. As a result of theanalysis 180 by the machine learning algorithm/process 176, infographics190 of analysis results and associated information are generated. Theinfographics 190 can include 3D models and tumor maps 192 of the tissueand/or a standard pathology report 194, both of which can be generatedautomatically with the ability to be communicated/displayed and viewedin real-time, and also uploaded to a patient's electronic medicalrecord.

II. Overview of Hardware and Software Arrangement

Having described an overview of an exemplary tissue specimen preparationtechnique and the settings, timing and results of the system and methodherein, the following is a more-detailed description of the structureand function of the system and method. FIG. 2 shows an arrangement 200for diagnosing conditions (e.g. various forms of cancer, including, butnot limited to, melanoma, squamous cell carcinoma, basal cell carcinoma(BCC), breast cancer, colon cancer, brain cancer, etc.) based upontissue sample slides prepared from patient tissue as part of a treatmentdiagnosis and regime or tissue sample slides prepared from animal modelsas part of a research study. The arrangement includes a computingprocessor 210 that can be any acceptable system, such as a server,laptop, PC, cloud computing environment, etc. The processor 210 receivesimage data from a variety of sources, including, but not limited to oneor more slide imagers 220, 222 that transform optical information from apart of, or whole, slides, via a microscope optics and associated imagesensor, into digital image data in a desired format. In a non-limitingexample Leica Aperia AT2 slide scanner, operating at 20× opticalmagnification, can be used to create 40K×40K pixel whole slide images(WSI). Six to eight tissue sections can be imaged in an exemplaryimplementation.

As described further below, the slide imager 220 is used in the field toimage patient slides for diagnosis in runtime, and this data 230 isthereby presented to the processor 210. One or more (other) slideimagers 222 can be part of a related or unrelated system that produces alarge volume of slide image data based upon various types of cellsand/or conditions. This data 232 is part of a training set that is inputto the processor 210 for use in construction a machine learning (AI)network (i.e. COMP_NET and BCC_NET) as described further below. Suchmachine learning can be based upon a variety of architectures andassociated computing algorithms/software—for example, a convolutionalneural network CNN. Note that the processor 210 is generallyrepresentative of one or more processing/computing devices that can beused in any of the stages of the overall system and method. In practice,one processor can be used to train the CNN, while another processorproduces final images and even another is used to operate a runtimeportal accessed by practitioners seeking to analyze one or more patientslides using the system and method. Hence, the term “processor” or“computing device” as used herein should be taken broadly to include oneor more discrete processors/computing devices used at one or more stagesof the over training and/or runtime operation of the system and method.Note also that further description related to the scanning of tissueslides, and handling data thereof, can be found in U.S. patentapplication Ser. No. 16/679,133, entitled SYSTEM AND METHOD FORANALYZING CYTOLOGICAL TISSUE PREPARATIONS, filed Nov. 8, 2019 by LouisJ. Vaickus, the teachings of which are incorporated by reference asuseful background information.

The processor 210 contains a plurality of functional processes(ors) ormodules. There is an image segmentation module/process(or) 252 thatallows both training and runtime slides to be broken into smallerfeature sets for reduction in processing overhead and/or to identifyspecific conditions within the data. This can allow for comparison, aswell as masking and other image processing operations described below. Atraining module/process(or) 254 controls the construction of the machinelearning network(s) 260, which is stored along with appropriateclassifiers, image data, etc. as shown. Likewise, a runtimemodule/process(or) 256 controls application of the machine learningnetwork 260 to runtime image data 230 to achieve diagnostic results.These results are handled by a result module/process(or) 258 that canpresent desired information graphically and/or textually as desired.

The process(or) 210 can be part of, or in communication with, acomputing device 270, which as described below can be any acceptablecomputing device or group of computing devices. The computing device 270can handle or manage system settings, user inputs and result outputs.The computing device 270 herein includes an exemplary graphical userinterface (GUI) having a display (e.g. a touchscreen 272, mouse 274 andkeyboard 276). The computing device 270 can interface with variousnetwork assets/data utilization devices, such as data storage, printers,display, robots, network ports, etc. Again, while the interface/displaydevice (computing device 270) herein is shown as a standalone PC orlaptop with separate keyboard and mouse, this can be representative ofany acceptable platform for entering, receiving and manipulatinginformation, including those with a single all-in-one functionality(e.g. a touchscreen display), such as found on a smartphone, tablet orminiaturized laptop.

It is recognized that machine learning has been used in multiple fieldsfor histologic diagnosis, but to date, has not been used for marginanalysis. The ability to rapidly analyze tissue margins and accuratelymap the tumor provides the ability to expand the use of en face/MMS typemargins into all areas of tumor resection in real-time. This system alsodecreases the amount of time it takes a pathologist to read an increasednumber of breadloafed paraffin sections thereby allowing for an increaseof margin analyzed while decreasing the amount of tissue required toread the tissue sections. In both instances machine learning providesthe opportunity to decrease the possibility of the pathologist missingthe presence of tumor at a margin and calling it a clear margin when infact the tumor involves the tissue margin.

III. System Overview and Operation

Reference is made to FIGS. 3A and 3B, which together graphically depictthe overall flow of data and operational procedure steps 300 inassociation with both training and runtime operation of the system andmethod herein. As shown, the workflow begins with the excision orresection of a solid mass of tissue from a patient in step 310. Then, instep 312, the tumor within the tissue is oriented and mapped. Next, instep 314, the tissue is frozen, sectioned (e.g. by en face, MMS,breadloafed etc.) and stained (substep 316) using an appropriatestaining method known to those of skill. Note, if performing paraffinembedded sections, step 314 would involve tissue processing, paraffinembedding, and, sectioning. Next, in step 320, a conventional or customslide scanner acquires one or more image(s) of the stained slide(s).These images are converted to digital image data—typically color and/orgrayscale—that can be stored and used for follow-on processing accordingto the system and method. In step 330, the image data (for example,whole slide data) can be transferred/downloaded to a local and/or cloudcomputing environment—for example, one or more server(s), that conductimage analysis on the slides based upon machine learning processesdescribed further below. These processes entail plurality of steps thatare described in further detail below. The steps include backgroundidentification 340 and tissue segment identification 342. Data/results350, including tile based images 352, region-adjacency or semanticrelationship graphs 354 and/or whole slides are fed to computingprocesses 360 that determine cancer versus no cancer (BCC_NET) 362and/or complete versus incomplete tissue images (COMP_NET) 364. Theoutputs of such processes 360 are then used to create predictions oftumor presence, size and shape (image 370). The predictions are used togenerate a model 380. This process, described further below, entailsmapping the sections of the tissue from the slides, stereotacticallyregistering the sections, and then creating a schematic 382 that can bepart of an overall infographic representation (described further below).As shown, the results can be displayed to a practitioner or otherinterested party with associated annotations and highlights via thedisplay 270. In this case, the display 270 depicts the relative locationof modeled tumorous tissue 390 versus the patient's anatomy 392. Resultscan also be displayed as a standard pathology report in cases that thealgorithm is used in post-operative margin analysis including but notlimited to paraffin embedded breadloafed sections.

Note that the illustrations herein are presented in grayscale. However,it is expressly contemplated that the various displays, heat maps,infographics, tissue region differentiations, etc. can bedepicted/displayed in a variety of colors symbolizing variousinformation and/or conditions. Likewise, various monochromatic shadings,fill patterns or other representation can be used instead of, or incombination with color representations to characterize the informationdisplayed herein. The implementation and/or use of such color-basedand/or shading-based representations on a display should be clear tothose of skill in the art.

Referring further to FIG. 4, the overall procedure 400 of processingimages is shown. The procedure 400 entails inputting raw slide images instep 410, followed by background deletion in step 420. This can beaccomplished using conventional or custom vision system tools, such asthose that analyze blob, edges and/or contrast. The images are thensubjected to connected component analysis (or a similar tool) to developa continuous representation of the region (which can be expressed as abinary image) in step 430. The images are then subjected to a labellingprocess in step 440. Finally, the trained COMP_NET and BCC_NET processesoperate on the images to generate results 450.

With reference now to FIGS. 5 and 6, the analysis of complete versusincomplete images by COMP_NET is described in further detail. Wholeslide images (WSI) are extracted and converted to NPY file format 500.Background deletion then occurs via (e.g.) Otsu thresholding. Connectedcomponents (objects separated by background) are then identified andfiltered by size. Each component is then evaluated by COMP_NET and givena “completeness” score. High risk components transmitted to surgeonand/or pathologist for evaluation from the results 600.

The analysis of cancer presence versus absence in tissue images byBCC_NET is now described in further detail. With reference to FIG. 7, inthe depicted exemplary image 700, the tumor (or high risk) regions 710and 720 are indicated. The areas are further highlighted in the image800 of FIG. 8. In operation, the initial model is trained. WSI areextracted and converted to NPY file format. Background deletion via(e.g.) Otsu thresholding then occurs. Connected components (objectsseparated by background) are identified and filtered by size. Eachcomponent is evaluated by BCC_NET and given a “carcinoma” score. Highrisk components transmitted to surgeon and/or pathologist for evaluationas part of the results 900 (FIG. 9).

The outputs from COMP_NET and BCC_NET are then applied to individualcomponents. The components are registered with MixMatch (a Custom GPUaccelerated gigapixel registration algorithm). The use of alternateregistration procedures should be clear to those of skill. A 3Dstereotactic infographic is then created and transmitted to surgeon. Asshown in the display 270 of FIG. 3B, clicking (for example, touchingwith a cursor 393 different segments in different symbolic colors,shades and/or fill patterns) on infographic 382, using an appropriateinterface device, brings up analyzed, highlighted WSI for that componentfor manual checking by the practitioner. The display 270 of FIG. 3B canalso feed (e.g., using appropriate communication/network andinterface/API arrangements) into a computer generated automatedpathology report with both text and infographic display of the presenceor absence of tumor at the specimen margin.

FIG. 10 further illustrates the overall operation of the system andmethod to create the color and/or shaded infographic 382 withinteractive regions that can be clicked upon to reveal furtherinformation to the user. As shown, tissue 1010 from the patient orresearch specimen is provided in the form of whole slides that areprepared from the tissue via margin excision, freezing or paraffinembedding, slicing and staining. This allows for use of WSI to provideimages 1020. These sections images have components identified usingvarious computing processes and are fed to both COMP_NET 1030 andBCC_NET 1032. These algorithms/processes respectively producecomplete/incomplete section results 1040 and positive or negative BCCresults 1042. The results are combined into an infographic 1050 that isdivided by sections 1052, which each display various types of tissuecomponents 1054. Again, the actual display can employ various symbolic(discrete) colors, which are represented in grayscale in this depiction.More generally, the contemplated use of discrete colors, patterns and/orshades in displays, some of which allow clicked accesses to enhancedrelevant data, can be termed “graphical differentiation” herein.

IV. Implementation Details

According to an embodiment, the above-described WSI are subjected to aseries of morphological and thresholding preprocessing algorithmsfollowed by background deletion and connected components analysis. Theabove-described, connected components analysis identifies and labelseach of the individual pieces of tissue present on the WSI. Typically,each individual component represents a single histological section(unless a section becomes fragmented, which rarely occurs). Thesubimages from each WSI are grouped according to a pathologist assignedcategory (complete or incomplete section and by presence or absence ofcarcinoma, e.g., BCC). The subimages are then downsampled by a factor of(e.g.) 10 using bicubic interpolation, separated in to Train, Validationand Test sets and used to train a neural network (many architectureswill be investigated). Model fit is evaluated by visual cluster analysisof latent feature vectors as well as by raw accuracy versus thevalidation set. The model hyperparameters are tuned until adequateperformance is achieved (e.g., >95%) versus the validation set. Themodels final accuracy is then calculated versus a test set. Two modelscan be trained via this methods, with one network to evaluate a tissuesection for completeness (COMP_NET) and one for evaluating for thepresence of malignant tissue, in this case BCC (BCC_NET). Note thatBCC_NET, instead of being trained on entire downsampled sections can betrained on 224×224 image subtiles from each section. The finishedBCC_NET can operate by passing a sliding window receptive field over aninput image with a high degree of overlap and summing BCC scores perarea to create a heat map (see FIGS. 8 and 9, and 370, 390 in FIG. 3B)of potential BCC locations.

Once these models are trained they are then added to a hybrid algorithm,which performs the following functions for all incoming images: (a)scanning, (b) preprocessing, (b) completeness evaluation (c) malignancyevaluation, (d) outputting information to the surgeon.

In operation, following slide cutting and staining, a local scanningdevice produces (e.g.) 20× scans as rapidly as possible (note thatcurrent Leica scanners (described above) can produce a 40× scan in 40seconds). Following scanning, the image can be saved to a “To Process”folder on an adjacent computing workstation. A process can be assignedto the incoming image, and the WSI will be loaded into RAM as a (e.g.)Numpy array for processing. The image is also normalized, deleting thebackground and performing connected component analysis to identifytissue sections. Each section can then be downsampled. The downsampledimages are accepted, and using COMP_NET, a binary prediction COMPLETE orINCOMPLETE is produced. AIM4 will pass the same subimages on to BCC_NETto render a binary prediction BCC or NEGATIVE. The evaluation can betransmitted to a display device (monitor 270 in FIGS. 2 and 3B) in theoperating room, which displays a representation of the resected tissueorganized by location in the operative site. Alternatively, if not beingperformed in real-time the evaluation can be used to generate anautomated pathology report to be transmitted to the practitioner. If allsections are complete for a margin, a series of overall (e.g.)green-shaded images (394 in FIG. 3B) can appear indicating the number oftissue sections analyzed representing the shape of the tissue segmentanalyzed (so the histotechnologist and surgeon can check for erroneousbreaks in the tissue). In the event of incomplete margins the sectionswhich were deemed incomplete can be displayed/represented in acontrasting color or graphic, such as red (396). Each segment can alsobe subtended with a BCC+ or BCC− title and/or areas of supposed BCC willbe highlighted/represented in another contrasting color or graphic, suchas purple (398). More generally, it can be seen how the cursor 393causes selection of a set of slides with emphasis on a selectedcolor/condition in slide 396 that is associated to that region. This isenlarged to a more detailed view in the (heat-map-based) tissue image390. This image is, itself, linked to the displayed photo of thepatient's anatomy 392—in this exemplary case, a region of the head inwhich a skin section has been excised, leaving an outline of skinsurrounding the underlying exposed tissue.

In various embodiments, preprocessing can also include further steps ofregistration and 3D reconstruction. Following connected componentanalysis, the image of each section is provided with an order, (e.g. 1,2, 3, 4, 5, etc.) based upon the order in which the histotechnologistcut the sections (which is typically the same every time). Briefly, thesecond section can be registered with the first section, the thirdsection with the second section, and so on, until a table ofregistration coordinates has been created. This table can specifytranslation, rotation and warping operations in order to most preciselyalign each section with its preceding section. Knowing that eachretained section is separated from its predecessor by a fixed number of5 micron-thick sections, the system can then construct a 3Dreconstruction of the original tissue. On the resulting displayeddiagram of the 3D reconstruction, the system can then highlight where in3D space any incomplete margins are located, and the location of BCC.

V. Services and Equipment

The above-described system and method can be offered to practitionersand facilities as an independent, (e.g.) subscription orper-use-fee-based service where each laboratory wanting to use thetechnology purchases or leases an appropriate whole slide scanner, alocal workstation (which will run the proprietary software used by thesystem and method) and a display for the operating room, which is linkedto the workstation. Arrangements can be made whereby such laboratoriesprovide additional training data from their runtime operations toimprove the training data/machine learning algorithms operated by thesystem and method. Any subscription or use-based service can includeappropriate secure Internet (or other network-based) communicationsprotocol layers and associated secure financial transaction processes.Appropriate passwords can be employed by users to authenticate access ina manner clear to those of skill.

In a client-service environment the following can be offered by theoperator of the system and method (in whole or part):

A. Intraoperative Margin Assessment (IMA)

-   -   IMA can provide 3D AI assessment of sectionings including but        not limited to, MMS, en face breadloaf (radial) margins in real        time to surgeons during operations via an intuitive infographic.

Requirements for Client:

-   -   1. Dedicated whole slide scanner, which can be purchased by the        customer or leased from the service.    -   2. Either (1) Internet upload speed of at least (e.g.) 25 Mbs        or (2) dedicated server purchased from service, preloaded with        licensed software. This device can be redundant, e.g. a double        server.    -   3. If the client has insufficient bandwidth for real time        uploads OR does not wish to purchase a server, their images will        be uploaded to our cloud service which can be strategically        deployed to servers physically near the client's location.    -   4. For the IMA cloud service, WSI can be uploaded to the        service's cloud servers and processed in the fastest possible        tier: STAT (can employ a very fast cloud computing instance).        This service can also be utilized for server customers if their        local hardware fails.

B. Post-Operative Margin Assessment (POMA)

-   -   POMA can provide 3D AI assessment of tissue margins including        but not limited to MMS, en face or breadloaf (radial) frozen        sections or paraffin embedded sections of surgical cases or        research studies for the purposes of increasing the number of        tissue sections analyzed, decreasing time for pathologist to        read slides, and increasing accuracy of pathologist read.

Requirements for client: as above for IMA

C. Retrospective Margin Assessment (RMA)

-   -   RMA can provide 3D AI assessment of tissue margins including but        not limited to MMS, en face or breadloaf (radial) frozen        sections or paraffin embedded sections of surgical cases or        research studies for the purposes of training, education and        (e.g.) legal/administrative cases.

Requirements for Client:

-   -   1. Access to physical slides or whole slide images (WSI).    -   2. Physical slides can be shipped to service's physical location        and scanned and uploaded to our cloud service at the customer        specified tier.    -   3. For the RMA cloud service, WSI can be uploaded to service's        cloud servers and processed in different services tiers,        depending on how fast the client wants results: STAT (can employ        a very fast cloud computing instance), EXPEDITE (can employ a        slower instance), ROUTINE (can employ the base GPU instance),        SLOW (will utilize spot instances), PRO BONO (can process during        downtime, on certain paid instances or in special        circumstances).    -   4. The client can receive back assessments of each margin and 3D        infographics as it would have appeared during a live operation.

C. Aggressiveness Risk Assessment (ARA):

-   -   ARA can analyze cases retrospectively for IMA, POMA and RMA        customers and provide a risk assessment for both local        recurrence (LR) and metastasis (M).

Requirements for Client:

-   -   1. Be part of either or ARA or RMA program.    -   2. Client can select either LR, M or LR/M (combo with (e.g.)        favorable price reduction).    -   3. Results can be computed either locally, or in the cloud        (cloud speed tier rates apply).    -   4. Results for LR and M can be provided as: Low Risk, Moderate        Risk, High Risk, or Indeterminate (for example, fees/charged for        Indeterminate can be waived).

VI. Conclusion

It should be clear that the above-described system and method provides ahighly robust and desirable tool for diagnosing and analyzing tissue forthe effectiveness of a treatment in the context of tumor resection. Thissystem and method operates with all common forms of histology andprovides a rapid, accurate, user-friendly, repeatable, and continuallyimproving result to practitioners and researchers.

The foregoing has been a detailed description of illustrativeembodiments of the invention. Various modifications and additions can bemade without departing from the spirit and scope of this invention.Features of each of the various embodiments described above may becombined with features of other described embodiments as appropriate inorder to provide a multiplicity of feature combinations in associatednew embodiments. Furthermore, while the foregoing describes a number ofseparate embodiments of the apparatus and method of the presentinvention, what has been described herein is merely illustrative of theapplication of the principles of the present invention. For example, asused herein, the terms “process” and/or “processor” should be takenbroadly to include a variety of electronic hardware and/or softwarebased functions and components (and can alternatively be termedfunctional “modules” or “elements”). Moreover, a depicted process orprocessor can be combined with other processes and/or processors ordivided into various sub-processes or processors. Such sub-processesand/or sub-processors can be variously combined according to embodimentsherein. Likewise, it is expressly contemplated that any function,process and/or processor herein can be implemented using electronichardware, software consisting of a non-transitory computer-readablemedium of program instructions, or a combination of hardware andsoftware. Additionally, as used herein various directional anddispositional terms such as “vertical”, “horizontal”, “up”, “down”,“bottom”, “top”, “side”, “front”, “rear”, “left”, “right”, and the like,are used only as relative conventions and not as absolutedirections/dispositions with respect to a fixed coordinate space, suchas the acting direction of gravity. Additionally, where the term“substantially” or “approximately” is employed with respect to a givenmeasurement, value or characteristic, it refers to a quantity that iswithin a normal operating range to achieve desired results, but thatincludes some variability due to inherent inaccuracy and error withinthe allowed tolerances of the system (e.g. 1-5 percent). Accordingly,this description is meant to be taken only by way of example, and not tootherwise limit the scope of this invention.

What is claimed is:
 1. A system for generating a model of tissue for usein diagnostic and surgical clinical and research procedures comprising:a machine learning process that receives whole slide images (WSI) oftissue removed from a patient or a research specimen and that determines(a) if each image of the WSI contain complete or incomplete tissuesamples and (b) if each image of the WSI contain tumorous tissue or anabsence of tumorous tissue; a reconstruction process that generates amodel of the tissue of the WSI with a mapping of types of tissuetherein; and a display process that provides results of the model foruse and manipulation by a user or pathology report.
 2. The system as setforth in claim 1 wherein the machine learning process is trained using aplurality of training slide images having a plurality of differingtissue types and arrangements.
 3. The system as set forth in claim 1wherein the training slide images are provided via at least one of alibrary of preexisting images of tissue from third parties andpreexisting images generated and stored by the user.
 4. The system asset forth in claim 1, further comprising a background removal processand a connected component process that pre-processes each of the slideimages prior to performing the machine learning process.
 5. The systemas set forth in claim 3 wherein the model defines differing sections ofthe tissue based upon each of the slide images and components of thetissue within each of the sections.
 6. The system as set forth in claim5 wherein the model includes an infographic that is arranged to allowthe user to access further information or images with respect to thesections or the components of the tissue via a user interface.
 7. Thesystem as set forth in claim 6 wherein the infographic is arranged todisplay a plurality of colors or other graphical representation thatcorrespond to different types of tissue.
 8. The system as set forth inclaim 6 wherein the machine learning process resides, at least in part,on a remote server accessed by the user via a network.
 9. The system asset forth in claim 8, further comprising an access control process thatauthenticates the user relative to the remote server and a billingprocess that generates financial transactions between the user and anoperator of the system with respect to operation of the system.
 10. Thesystem as set forth in claim 1 further comprising a whole slide imagerarranged to read each WSI prepared by the user and provide the slideimages therefrom to the machine learning process.
 11. The system as setforth in claim 1 wherein slides associated with the WSI are preparedusing frozen sections or paraffin embedded sections of the tissue. 12.The system as set forth in claim 11 the tissue is grossed using an MMS,en face or breadloaf (radial) technique.
 13. The system as set forth inclaim 1 wherein the pathology report includes a readout that is adaptedto be uploaded to a chart of the patient.
 14. A method for generating amodel of tissue for use in diagnostic and surgical clinical and researchprocedures comprising the steps of: receiving whole slide images (WSI)of tissue removed from a patient or a research specimen and determining,using a machine learning process, (a) if each image of the WSI containcomplete or incomplete tissue samples and (b) if each image of the WSIcontain tumorous tissue or an absence of tumorous tissue; a generating amodel of the tissue of the WSI with a mapping of types of tissuetherein; and providing results of the model for use and manipulation bya user or pathology report.
 15. The method as set forth in claim 14,further comprising, training the machine learning process using aplurality of training slide images having a plurality of differingtissue types and arrangements.
 16. The method as set forth in claim 14,further comprising, pre-processing each of the slide images prior to thestep of determining, using the machine learning process, by performing abackground removal process and a connected component process.
 17. Themethod as set forth in claim 14 wherein the model defines differingsections of the tissue based upon each of the slide images andcomponents of the tissue within each of the sections.
 18. The method asset forth in claim 14 wherein the step of providing includes displayingan infographic that is arranged to allow the user to access furtherinformation or images with respect to the sections or the components ofthe tissue via a user interface.
 19. The method as set forth in claim18, further comprising, performing an access control process thatauthenticates the user relative to the remote server and performing abilling process that generates financial transactions between the userand an operator of the system with respect to operation of the system.20. The method as set forth in claim 14 wherein slides associated withthe WSI are prepared using frozen sections or paraffin embedded sectionsof the tissue, and the tissue is grossed using an MMS, en face orbreadloaf (radial) technique.